Cluster-wise license information replication

ABSTRACT

System and methods for evaluating license information in a computer cluster are described. An example method may include retrieving a cluster-license-info from a peer-node selected from a plurality of cluster nodes in the computer cluster, wherein the cluster-license-info contains a plurality of peer-license-info collected from the plurality of cluster nodes, receiving a revised cluster-license-info generated by detecting license violations in the cluster-license-info, and transmitting the revised cluster-license-info to the peer-node.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a divisional application of U.S.Nonprovisional application Ser. No. 16/439,729, filed Jun. 13, 2019. TheU.S. Nonprovisional Application, including any appendices or attachmentsthereof, is incorporated by reference herein in its entirety.

BACKGROUND

In a clustered computing environment (“computer cluster”), validatingand enforcing software licenses often requires storing and managing thelicense information of the cluster nodes in a centralized node. In anend-user's computer cluster site, a software vendor may need to verifywhether there is any illegal or unauthorized usage of its software bycommunicating either with each cluster node, or by obtaining a copy ofthe license information from the centralized node. However, a providerof the end-user site may be uncomfortable exposing its critical ITproperties including license information to the software vendor.Therefore, software vendors may have to dispatch field operators to theend-user site and try to detect any license abuses while performingon-site support. Still, in this centralized license informationconfiguration, it is hard for the field operators to convince theend-user site to allow access to its centralized node.

In another conventional license management methodology, licensemanagement software (e.g., the RLM from reprise) may be used to managelicense information for the computer cluster. This license managementsoftware may be composed of a specific license agent, license files andclient library. However, all the software to be protected by the licenseagent must be linked with the client library. Further, during runningtime, the license agent must directly interact with the software tovalidate the license in-use. Thus, such a method may bring morecomplexity and costs to the management of any software. And it's toughfor field operators to distinguish a valid license from an illegal one.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of a system configured to providecluster-wise license information replication and validation, accordingto one or more embodiments of the present disclosure.

FIGS. 2A, 2B, and 2C illustrate example communications among the clusternodes during cluster-wise multicast license information aggregation,according to one or more embodiments of the present disclosure.

FIG. 3 shows a license inspection and enforcement process, according toone or more embodiments of the present disclosure.

FIG. 4 shows a flow diagram illustrating a process for replicating andenforcing license information in a computer cluster, according to one ormore embodiments of the present disclosure.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented here. It will be readily understood that the aspects of thepresent disclosure, as generally described herein, and illustrated inthe Figures, can be arranged, substituted, combined, and designed in awide variety of different configurations, all of which are explicitlycontemplated herein.

The present disclosure is related to apparatus and methods to improvethe efficiency of illegal license detection in mission-critical andhigh-value cluster software. Illegal use of software without properlicenses not only causes huge revenue losses for software vendors, butalso hurts or even destroys software industry in the long run. Formission critical and high value software, on-site service requeststhrough software vendors' own field operators may present a good chanceto detect illegal software license abuse. However, license managementcomponent of the cluster software is usually deployed in a centralizedlicense management node, and any access by external field operators isusually denied. In other situations, the license information may spreadacross multiple nodes in the computer cluster, making evaluatingsoftware license one node at a time tedious and near-impossible. Thus,many potential license abuse cases may never be detected if the servicerequest is based on one valid license, while the other nodes do not haveproper licenses.

The present disclosure relates to the replicating and enforcing oflicense information of mission critical and high value software acrossthe whole computer cluster. In particular, the present disclosure uses amulticast-based cluster-wise license information transfer protocol torapidly and robustly replicate license information to each cluster nodein the computer cluster in a scalable way, and use encryption to protectthe replicated license information from malicious attacks. In addition,a field operator may use a smartphone application to access any singlenode in the computer cluster, and quickly detect license abuse in thewhole computer cluster.

FIG. 1 illustrates a block diagram of a system configured to providecluster-wise license information replication and validation, accordingto one or more embodiments of the present disclosure. In FIG. 1 , thesystem may include a computer cluster 140 and a license validator 130.The computer cluster 140 and/or the license validator 130 may beimplemented using one or more clouds based on a cloud computingenvironment 160. During operation, a set of license agents 141 may beconfigured to manage the various licenses in the computer cluster 140.And a field operator 110 may utilize a license review device 115 toretrieve and review the license information associated with the computercluster 140.

In some embodiments, the computer cluster 140 may include a set ofcomputer nodes 150 loosely or tightly connected via a cluster network.Each of the computer nodes 150 may be referred to as a “cluster node”,“computer node”, or “node.” Further, each cluster node 150 may be acomputing device or an electronic device capable of operating jointly orindependently. For example, a cluster node 150 may be a personalcomputer, a server computer, a smartphone, a consumer device such as atelevision, a vehicle control device installed on a car, or any otherdevice that has computational and networking capabilities.

In some embodiments, the cluster nodes 150 in the computer cluster 140may be integrated together so that they can be viewed as a single systemfrom an external perspective. The cluster nodes 150 may be connected toone or more cluster networks, so that one cluster node 150 may be ableto directly or indirectly communicate with any other cluster nodes 150in the computer cluster 140. The cluster network may be implementedusing one or more network communication protocols that allow networkmessages to be broadcasted, relayed, and otherwise delivered, and may bean Ethernet, Local Talk, Straight Bus, FDDI Ring, Token Ring, ATM, StarNetwork, or peer-to-peer network.

In some embodiments, the cluster node 150 may be installed with theirrespective instances of operating system and software, and may performthe same or different tasks according to cluster configurations.Multiple cluster nodes 150 may be deployed with redundancy,fault-tolerance, or failover capabilities, in order to improveperformance and availability over that of a single computer. Thus, thecomputer cluster 140 may have a wide range of applicability anddeployment, ranging from small business clusters with a handful of nodesto supercomputers within hundred-thousand of nodes.

In some embodiments, each of the cluster nodes 150 may be installed withcertain software applications that are subject to various softwarelicenses. A “license” may be a legal instrument governing the terms andconditions related to the use or distribution of a specific software orapplication. Types of license may include, without limitation,device-based, user-based, individual, group, periodical, subscription,and/or perpetual. For example, a “device-based license” may beassociated with a specific device (or CPU), and may only allow thisspecific device to execute the software regardless of the number ofusers that can access the software through this specific device. A“user-based license” may be associated with a specific user, and mayonly allow the specific user to use the software regardless of fromwhich devices the user may access the software. From a quantityperspective, an “individual license” may be used on one device or by asingle user; a “group license” may allow a fixed number of concurrentdevices or users to use the software at the same time; a “periodicallicense” may allow the usage of software within a specified time period;a “subscription license” may allow the usage of software within arenewable time period; and a “perpetual license” may allow the usage ofsoftware indefinitely. In some instance, a software license may beconfigured with multiple license types. For example, a softwareapplication may require a license that is user-based, group, andperiodical.

In some embodiments, the computer cluster 140 may be configured with oneor more multicast-based license agents 141 to manage the various typesof licenses that are activated in the cluster nodes 150. Specifically,each of the cluster nodes 150 may be installed with a correspondinglicense agent 141, and these license agents 141 may communicate amongthemselves via the cluster network for the replicating and enforcing ofsoftware licenses in the computer cluster 140. In other words, eachlicense agent 141 may monitor the status and usage of the softwarelicenses in the specific cluster node 150 on which the license agent 141is installed. The license agents 141 may generate license informationfrom the cluster nodes 150, and may further propagate, aggregate, andsynchronize the license information collected from these cluster nodes150.

In some embodiments, the license agents 141 among the cluster nodes 150may use a quorum-based election algorithm to dynamically elect (144) oneof the cluster nodes 150 to act as a license-info aggregation anddistribution center. The elected cluster node may be referred to as acore-node 154, which may be responsible for collecting and distributioncluster-wise license information among all the cluster nodes 150. Therest of the cluster nodes 140 that are not elected core-node 154 may bereferred to as “cluster member nodes” or “peer-nodes” (e.g., peer-node152). If the core-node 154 is no longer accessible due to hardware,software, or network failures, the license agents 141 among thepeer-nodes may dynamically elect another one of the cluster nodes 150 toserve as the new core-node 154. In this case, all the other clusternodes 150 that are not the core-node 154 may be deemed peer-nodes. Whenthe original core-node is accessible again, it may detect a newcore-node 154 in the computer cluster 140, and may adjust itself to be apeer-node. For convenience purposes, all license-related communicationsamong the cluster nodes 150 may be assumed being performed by thecorresponding license agents 141 installed on these cluster nodes 150.

In some embodiments, the cluster nodes 150 may utilize multicastcommunications 142 to broadcast and transmit license-related networkmessages. Specifically, the core-node 154 may broadcast acollection-request message to the peer-nodes. Upon receiving thiscollection-request message, each peer-node may continuously monitor andcollect its own license information, and generate a license-infocorresponding to the peer-node. Specifically, “license-info” may containa license-configuration and a license-usage-data. The“license-configuration” may specify the types of licenses (e.g.,user-based, group, etc.) as well as the status of the licenses (e.g.,activated, expired, un-licensed, etc.). The “license-usage-data” mayrecord the device information associated with the cluster node thesoftware is installed on, as well as the number of users, the number ofsimultaneous users, the user IDs, and the access info (e.g., timestamp)associated with software usage. Thus, license-info from the peer-node152 or the core-node 154 may be referred to as “peer-license-info”.

In some embodiments, the peer-nodes may transmit the generatedpeer-license-info to the core-node 154, and the core-node 154 mayaggregate and synchronize the multiple peer-license-info (including itsown license-info) into a “cluster-license-info”, which may contain thelicense-configurations and license-usage-data of all the cluster nodes150. Afterward, the core-node 154 may distribute copies of thecluster-license-info to each of the peer-nodes. The above multicastcommunications 142 among the license agents 141 may be implemented usingany communication protocols which can transmit/deliver a copy of networkmessage from one cluster node to multiple cluster nodes, asynchronouslyor synchronously, over wired or wireless network.

In some embodiments, a field operator 110 may be a person who needs toaccess a particular peer-node 152 in the computer cluster 140 for aparticular purpose (e.g., maintaining and servicing the hardware andsoftware on this particular cluster node). The field operator 110 mayalso access any one of the cluster nodes 150 or the core-node 154.During inspection, the field operator 110 may utilize a license reviewdevice 115 to interact with the peer-node 152. The license review device115 may be a portable device (e.g., laptop or smartphone) that cancommunicate with the peer-node 152 using wired or wireless networkcommunications (e.g., WIFI, Bluetooth, mobile network communications).For example, the license review device 115 may be a smartphone equippedwith at least one camera, and can connect to carrier wireless network.Further, the smartphone may be installed with a “smartphoneapplication”, or “app”, to interact with the peer-node 152.

In some embodiments, the license review device 115 may retrieve from thepeer-node 152 the cluster-license-info 120 stored therein, and transmitit to a license validator 130 for license authentication and validation.Specifically, the license validator 130 may be a remote server that canevaluate the cluster-license-info 120 to determine whether there are anylicense violations in any of the cluster nodes 150. License violationsmay include license-expiration violation, a license-abuse violation, ora no-license violation. License-expiration may refer to the expirationof any previous valid license; license-abuse indicates there might behacked licenses or unauthorized or misused licenses. For example, havingmore concurrent users than the license allowed may be a form of licensemisuse. No-license means there is no valid license installed.

In some embodiments, the license validator 130 may generate a “licenseevaluation status”, which may identify those cluster nodes 150 havinglicense violations, and transmit this license evaluation status to thelicense review device 115. The field operator 110 may then takeappropriate actions to enforce the licensing situations in the computercluster 140. Further, the license validator 130 may generate a “revisedcluster-license-info” based on the received cluster-license-info 120,and transmit the revised cluster-license-info to the license reviewdevice 115 for license enforcement. The revised cluster-license-info maycontain a set of enforcement actions that can be invoked to enforcelicenses in the computer cluster 140. For example, the revisedcluster-license-info may contain enforcement actions toactivate/deactivate certain licenses in some of the cluster nodes 150,and/or adjust the number and types of licenses for other cluster nodes150 accordingly.

In some embodiments, the license review device 115 may transmit therevised cluster-license-info to the peer-node 152, which may in turnpropagate this revised cluster-license-info to all the cluster nodes 150using multicast communications 142. For each of the cluster nodes 150,its license agent 141 may process the revised cluster-license-info itreceived, and perform any enforcement actions that are applicable to thepresent cluster node. Thus, the license agents 141 mayactivate/deactivate or adjust the licenses to enforce license compliancein the computer cluster 140

Thus, by using a light-weight replication protocol and efficientlydistributing cluster-wise license information to every cluster node 150,the above approach may enable field operator to efficiently validate andenforce software license in a computer cluster via a simplifiedsmartphone application. And additional encryption techniques may furtherpreserve the integrity of cluster-wise license information.

In some embodiments, the license validator 130 and/or the computercluster 140 may be implemented by one or more clouds. A “cloud” may be anetwork-based computing architecture that provides shared pools of cloudresources on demand. The cloud may also be implemented based on a cloudcomputing environment 160, which may contain, among other components,one or more virtual machines (VMs) 171-174 and/or physical machines(“hosts”) 181. Thus, the cloud may be constructed using multiple hosts181, and/or multiple VMs 171-174 that are created based on some of thehosts 181. Further, each of the cluster nodes 150 in the computercluster 140, as well as the license validator 130, may be either a VMs171 or a host 181.

In some embodiments, the cloud may include a cloud manager/director (notshown in FIG. 1 ) configured for implementing the various cloudfunctionalities such as resource pooling, resource allocating,high-availability, and automation, etc. In some embodiments, the cloudmay be constructed using products such as VMWARE® vCloud, and the cloudmanager may be implemented using a VMWARE vRealize Suite. The cloud 140may also be configured with a VMWARE VSPHERE server. Alternatively, thecloud 140 may be constructed using any commercial cloud products, suchas OpenStack® Cloud, and/or AMAZON® S3 Cloud.

In some embodiments, the cloud computing environment 160 may include oneor more virtual machine execution space 170. Each VM execution space 170may contain multiple VMs 171, 172, 173, and 174 configured to hostvarious applications. Further, the cloud computing environment 160 mayinclude a VM manager or hypervisor (such as VMware ESX® basedhypervisor, or Microsoft® Hyper-V® virtualization, all of which are notshown in FIG. 1 ) to create the VMs 171-174 based on the hardwareresources 180. The hardware resources 180 may include one or more hosts181, each of which may be a physical computer system having a “physicalhardware platform” (e.g., an x86 architecture platform). The hypervisormay be configured to construct a “virtual hardware platform” for the VMs171-174 based on the host 181′s physical hardware platform.

In some embodiments, to support the physical hardware platform, thehardware resources 180 may include various “physical hardwarecomponents” such as, without limitation, one or more physical CentralProcessing Units (CPUs) 182, physical memory 183, physical NetworkInterface Card (NIC) 184, physical storage (e.g., hard drive) 185,and/or additional electronic circuit components (all of which may besupplied by one or more hosts 181). Each of CPU(s) 182 may be configuredto execute instructions that perform one or more operations describedherein. The instructions can be stored in the memory 183, storage 185,or any other memory in the hosts 181 (e.g., cache memory) that, whenexecuted by the CPU 182, cause the CPU 182 to perform certain operationsas described herein. The memory 183 may be non-transitory storage mediumconfigured to store various information, and may include, e.g., randomaccess memory (RAM), read-only memory (ROM), or a combination thereof.The NIC 184 may include one or more network adapters. And the storage185 may include local storage devices, such as hard disks, flash memorymodules, solid state disks, optical disks, and the like. Storage 185 canalso include interface(s) configured for communication with one or morenetwork data storage systems, such as host bus adapter(s) configured forcommunication with a storage array network (SAN) or network-attachedstorage (NAS), as well as other network data storage systems.

In some embodiments, based on the hardware resources 180, the VM managermay configure a “virtual hardware platform” for the VMs 171-174 with oneor more “virtual hardware components” such as, without limitation, oneor more virtual CPUs, virtual memory, virtual storage, virtual NIC,and/or additional virtual components (not shown in FIG. 1 ). With helpsfrom the VM manager, the virtual hardware components may emulate thebehaviors and the computing capabilities of the corresponding physicalhardware components, thereby allowing the VMs 171-174 to function as ifthey were physical hosts 181.

In some embodiments, the license agent 141 (and the various modulescontained therein) may be implemented using hardware modules or asoftware system. For example, the license agent 141 may be implementedusing a programmable logic device (e.g., Field-programmable gate array,or “FPGA”), which may be an electronic module or digital circuit thatcan be field-configurable after manufacturing. In other embodiments, thelicense agent 141 may be a network-enabled hardware module installed ina cluster node 150 and allows external network communications.

FIGS. 2A, 2B, and 2C illustrate example communications among the clusternodes during cluster-wise multicast license information aggregation, inaccordance to one or more embodiments of the present disclosure. InFIGS. 2A-2C, various modules/components, including core-node, clusternodes, and license review device, may correspond to their respectivecounterparts in FIG. 1 . For example, the core-node and peer-nodes inFIGS. 2A-2C may correspond to the core-node and peer-nodes of FIG. 1 .

In FIG. 2A, a computer cluster may perform a “cluster-initialization”process to collect and replicate all the license information among thecluster nodes 220. The cluster-initialization process may be activatedafter the initial deployment (including installation and configuration)of the software in the computer cluster. In this case, the core-node 210may wait for the completion of the initial configurations of thesoftware in all the cluster nodes 220 before starting thecluster-initialization process. In some instances, the core-node 210 maybe in charge of all the peer-nodes' software deployment, and so it knowswhen the software deployment process is completed. Alternatively, thecore-node 210 may monitor the peer-nodes 220's software deploymentprogress.

In some embodiments, after the initial deployments of the software inthe computer cluster, the core-node 210 may transmit (211) acollection-request message (“COLLECTION_REQ”) to each of the peer-nodes220 in the computer cluster, requesting all peer-nodes 220 to collecttheir respective license-info from those software/hardware that aresubject to certain license regulations. Specifically, the license agentsinstalled on all the cluster nodes may bind to a specific networkcommunication port to listen to multicast messages. Thus, once thecore-node 210 broadcasts the COLLECTION_REQ message through this networkcommunication port, all the peer-nodes 220 that are listening to thisport may receive this collection-request message.

In some embodiments, after receiving the collection-request message fromthe core-node 210, each of the peer-nodes 220 may collect its respectivelocal license-info (including license-configuration andlicense-usage-data) and generate a corresponding peer-license-info.Optionally, the license agent may retrieve the hardware feature tags(e.g., mainboard serial number, network identifier, firmware/bios uniquetag, etc.) associated with the peer-node, and use the hardware featuretags to encrypt the peer-license info. Afterward, in response to thecollection-request message, each peer-node 220 may transmit (212) acollection-response message (“COLLECTION_REP”) to the core-code 210. Thecollection-response message may contain the peer-license-info.

In some embodiments, if the core-node 210 does not receivecollection-response messages from all the peer-nodes 220, it may eitherrestart the above collection process until the peer-license-info fromall the peer-nodes 220 are received successfully. Alternatively, thecore-node 210 may transmit (211) an individual collection-requestmessage to those peer-nodes 220 that it did not receive theirpeer-license-info, and wait for those peer-nodes 220 for theircollection-response messages. The core-node 210 may also collect its ownlicense-info.

In some embodiments, based on the peer-license-info received from allthe peer-nodes 220 and the license-info from its own, the core-node 210may aggregate and generate a cluster-license-info that containscluster-wise license information from all the cluster nodes in thecomputer cluster. Afterward, the core-node 210 may transmit (213) to allthe peer-nodes 220 a distribution message (“DISTRIBUTE”) which includesthe cluster-license-info. Upon receiving the distribution message, eachpeer-node 220 may transmit (214) an acknowledge message(“DISTRIBUTE_ACK”) to the core-node 210, indicating that it has receivedand stored the cluster-license-info locally. After receiving theacknowledge messages from all the peer-nodes 220, the aboveinitialization process is completed, and each cluster node in thecomputer cluster maintains a copy of the cluster-license-info.

In some embodiments, if the core-node 210 does not receive ACK messagesfrom all the peer-nodes 220, it may restart the above collection processuntil it receives peer-license-info from all the peer-nodes 220.Alternatively, the core-node 210 may transmit (213) an individualdistribute message to those peer-nodes 220 that did not deliveracknowledge messages, and wait for those peer-nodes 220 for theirDISTRIBUTE_ACK responses. If the number of core-node 210 retrying theabove retransmission process surpasses a specific threshold (e.g., 5times), the license agent on the core-node 210 may transmit an alarm tothe sysops through administrative communication channels (e.g., mailboxor message applications).

In some embodiments, the cluster nodes may utilize a cluster-wise“epoch” value to keep track of the different versions ofcluster-license-info being distributed to the cluster nodes from time totime. For example, the epoch value may be a monotonically-increasing64-bit unsigned number, and may be stored in a persistent location(e.g., a database or a secured file) in the core-node 210 and all thepeer-nodes 220. The epoch value may be increased in each round ofcluster-license-info distribution. In other words, before distributingthe cluster-license-info to the peer-nodes 220, the core-node 210 mayfirst increment the epoch value, embed the epoch value along with thecluster-license-info in the distribution message, and then transmit(213) the distribution message to each of the peer-nodes 220. After around of cluster-license-info distribution, all the cluster nodes in thecomputer cluster, including the core-node 210 and the peer-nodes 220,will receive the same epoch value and the same cluster-license-info, andmay store the received epoch value and cluster-license-info in theirlocal persistent storage.

In some embodiments, if during subsequent communications, a cluster-nodereceives another version of the cluster-license-info along with anotherepoch value, it may compare the received epoch value with its locallystored epoch value. If the received epoch value is the same as orsmaller than the stored epoch value, it means the receivedcluster-license-info is either the same as the storedcluster-license-info, or is an older version of the cluster-license-infothan the stored cluster-license-info. In this case, the cluster node maychoose to ignore the received cluster-license-info.

In some embodiments, after sending (211) out multicastcollection-request message, the core-node 210 may start a timeoutcounter to wait for responses from all the peer-nodes 220. If thecore-node 210 does not receive response messages from some of thepeer-nodes 220 within the timeout counter period, it may either cease towait for additional response messages, or resend (211) a specificcollection-request message to certain peer-nodes 220 which existencesthe core-node 210 is aware of. Likewise, after transmitting (213) thedistribution message, the core-node 210 may start a timeout counter towait for acknowledge messages from the peer-nodes 220, and may eitherresend (213) the distribution message, or stop waiting for acknowledgemessages.

In some embodiments, before sending (212) the peer-license-info to thecore-code 210, each peer-node 220 may compress and encrypt thepeer-license-info using the peer-node's hardware feature tags. This way,even though the core-node 210 receives the peer-license-info from allthe peer-nodes 220, without the proper decryption keys, the core-node210 can only generate a cluster-license-info based on the encryptedpeer-license-info, but cannot decrypt the peer-license-info. Such anapproach may ensure that the license information in the computer clusterwould not be hacked or abused.

In FIG. 2B, after the computer cluster completed the aboveinitialization process, the core-node 230 may perform a “licenseupdating” process to update the license information replicated in thecluster nodes 240 when an update event 243 occurred in the computercluster. The “update event” 243 may include any situations when newlicense information is added or when existing license information ischanged in the computer cluster. For example, after the initializationprocess as shown in FIG. 2A is completed, a new peer-node with itsdistinctive license configuration may be added into the computercluster; or an existing peer-node may have its license configurationchanged (e.g., license added or expired). In these situations, the newpeer-node or the existing peer-node may be referred to as an “updatedpeer-node” 241, and the adding of the peer-node or the updating of thelicense information may be deemed an update event.

In some embodiments, the core-node 230 may be able to detect an updateevent 243 at the updated peer-node 241. Specifically, when the updatedpeer-node 241 is a new cluster node just added into the computercluster, the new cluster node may identify and communicate with thecore-node 230 by querying all the other peer-nodes 240. In this case,the core-node 230 may discover this new cluster node, and may be able todetermine this update event 243 indicating that new license informationfrom the updated peer-node 241 needs to be included into thecluster-license-info. Alternatively, the core-node 230 may periodicallyscan the computer cluster for any newly added peer-node 241.

In some embodiments, the updated peer-node 241 may be an existingcluster node in the computer cluster, and the update event 253 may be alicense-related event, such as the adding of new licenses or theexpiring of existing licenses at the updated peer-node 241. In thiscase, the license agent of the updated peer-node 241 may transmit theupdate event 253 to the core-node 230, informing that the licenseinformation in the computer cluster needs to be updated.

In response to an update event 243 from an updated peer-node 241, thecore-node 230 may transmit (231) a collection-request message(“COLLECTION_REQ”) to the updated peer-node 241, requesting it tocollect its license-info from those software/hardware that are regulatedby certain licenses. In response to the collection-request message, theupdated peer-node 241 may collect its peer-license-info, and maytransmit (232) a collection-response message (“COLLECTION_REP”) to thecore-code 230. Optionally, the peer-license-info contained in thecollection-response message may be compressed and encrypted.

In some embodiments, after receiving the peer-license-info from theupdated peer-node 241, the core-node 230 may aggregate thepeer-license-info into an updated cluster-license-info, and mayincrement the previously maintained epoch value. The core-node 230 maythen transmit (233) a distribution message (“DISTRIBUTE”) to all thecluster nodes 240 (including the updated peer-node 241). Thedistribution message may include the updated cluster-license-info andthe newly incremented epoch value. Upon receiving the distributionmessage, each cluster nodes 240 may transmit (234) a acknowledge message(“DISTRIBUTE_ACK”) to the core-node 230, indicating that it has receivedand stored the updated cluster-license-info. After receiving theacknowledge message from all the other cluster nodes 240, each clusternode in the computer cluster maintains a copy of the updatedcluster-license-info and a copy of the newly incremented epoch value.

In some embodiments, the updated peer-node 241 may also represent apeer-node which has been included in the previously publishedcluster-license-info, but subsequently has license changes. Licensechanges may include installation and activation of licenses, terminationand expiration of licenses, and/or changes in scope and coverage of theexisting licenses. In this case, the license agent of the updatedpeer-node 241 may communicate with the license agent of the core-node230, notifying the update event 243, and the core-node 230 may performthe above licensing updating process to replicate the changes to thecluster nodes 240 in the computer cluster.

In some embodiments, the core-node 230 may generate apartial-cluster-license-info, which only contains the peer-license-inforelated to the updated peer-node 241. In other words, comparing to aregular cluster-license-info which contains peer-license-info from allthe cluster nodes, the “partial-cluster-license-info” does not containany peer-license-info from any other peer-nodes 240 that do not havelicense changes. In this case, the core-node 230 may transmit a regularcluster-license-info to the updated peer-node 241, and distribute (233)the partial-cluster-license-info (along with a newly incremented epochvalue) to the peer-nodes 240 that do not have license changes. Afterreceiving the partial-cluster-license-info, the peer-nodes 240 that donot have license changes may generate a new version of thecluster-license-info by combine/merge the partial-cluster-license-infowith its own copy of the existing version of cluster-license-info. Forexample, the peer-nodes 240 may remove from the existing version of thecluster-license-info any license-info related to the updated peer-node241, and then insert the partial-cluster-license-info into the existingversion to create the new version of the cluster-license-info.Afterward, the peer-nodes 240 may store the new version of thecluster-license-info and the new epoch value in its local persistentlocation.

In FIG. 2C, the core-node 250 may perform a “license removing” processto update the license information replicated in the cluster nodes 260when a remove event 253 occurred in the computer cluster. The “removeevent” 253 may include any situations when license information ischanged due to the removal of a cluster node from the computer cluster.For example, the peer-node 261 may be physically removed from thecomputer cluster due to hardware/software failure or retirement, or maybe logically removed from the computer cluster due to configuration. Inthese situations, the affected peer-node may be referred to as a“removed peer-node” 261, and the removal of the peer-node 261 may bedeemed a remove event.

In some embodiments, the license agent of the removed peer-node 261 mayfirst allocate the core-node 250 in the computer cluster, and thentransmit a message to the core-node 250 indicating its intent to beremoved from the computer cluster. Alternatively, the core-node 250 mayperiodically scan the computer cluster for any peer-nodes that are nolonger accessible for a predetermined amount of time. If the core-node250 detects a large number of cluster nodes 260 not-accessible via thecluster network, it may determine that a large scale network malfunctionmay be happening, or the connectivity issue may be due to itself. Inthis case, the core-node 250 may either wait a while before checking thenetwork condition, or determine that it is no longer the core-node ofthe computer cluster, as the rest of the cluster nodes 260 may alreadyelect another cluster node to serve as the new core-node.

In some embodiments, in response to the identifying of a removedpeer-node 261, the core-node 250 may adjust its copy ofcluster-license-info to remove license-info associated with the removedpeer-node 261, and may generate an updates cluster-license info.Afterward, the core node 250 may transmit (251) a distribution messageto all the peer-nodes 260 (excluding the removed peer-node 261). Thedistribution message may include the updated cluster-license-info and anewly incremented epoch value. Upon receiving the distribution message,each cluster nodes 260 may transmit (252) an acknowledge message to thecore-node 250, indicating that it has received and stored the updatedcluster-license-info and the newly incremented epoch value. Afterreceiving the acknowledge message from all the other peer-nodes 260,each cluster node in the computer cluster maintains a copy of theupdated cluster-license-info which no longer includes license-info ofthe removed peer-node 261. In addition, the core-node 250 may stopdelivering any new license-related messages to this removed peer-node261.

In some embodiments, the licenses on a peer-node 261 may become expiredor terminated. In this case, the license agent on the peer-node 261 maytreat the peer-node 261 as an updated peer-node, and the core-node mayupdate the cluster-license-info based on the license update process asshown in FIG. 2B. Alternatively, the license agent on the peer-node 261may remove/uninstall the software associated with the expired/terminatedlicenses, and treat the peer-node 261 as a removed peer-node 261, andthe core-node 250 may perform a license removal process as shown in FIG.2C. Optionally, the removed peer-node 261 may delete the locally storedcluster-license-info and the epoch value.

FIG. 3 illustrates a license inspection and enforcement process,according to one or more embodiments of the present disclosure. In FIG.3 , various system and components may correspond to their respectivecounterparts in FIG. 1 . In some embodiments, a field operator 110 maybe able to interact with a peer-node 152 to inspect/enforce licenses inthe whole computer cluster. For example, the field operator 110 mayreceive a service request from a specific software vendor, indicating aproblem or bug issue related to the software installed in the computercluster. The field operator 110 may then interact with a peer-node 152,which may be any one of the cluster nodes in the computer cluster,either by remotely log-in, or by physically go on-site to access thepeer-node 152. Thus, the field operator 110 may be able to validate thecluster-wise license information of the whole computer cluster during anon-site service of a single cluster node.

In some embodiments, the field operator 110 may have access to thelicense agent installed on the peer-node 152, and utilize a licensereview device 115 to retrieve and review the cluster-license-info 120stored on the peer-node 152. Specifically, the license agent installedon the peer-node 152 may be configured to generate one or more QR codeimages (e.g. QR code 310) for the license review device 115. The licenseagent may pre-generate and store the generated QR code 310 on thepeer-node 152. Alternatively, license agent may dynamically generate theQR code 310 when the field operator 100 invokes the license agent.

In some embodiments, the QR code 310 may include a locator, identifier,or tracker (e.g., URL path or database access ID) to direct the licensereview device 115 to the cluster-license-info 120 stored in thepeer-node 152. Alternatively, the whole cluster-license-info may beembedded into the QR code 310. When the cluster-license-info containsencrypted data, the license agent may utilize a binary-to-text encodingmodule (e.g., base64 or uuencode) to generate text-formatted data forthe QR code 310. Further, the license agent may also include in the QRcode 310 hardware feature tags of the peer-node 152 as well asencryption keys used for encrypting the cluster-license-info.

In some embodiments, the application or app installed in the licensereview device 115 (e.g., a smartphone) may be configured to determinethe integrity of the cluster-license-info and detect any license abusesin the computer cluster. Once the QR code 310 is displayed on thepeer-node 152, the field operator 110 may use the smartphone app to readthe QR code 310, and decode the information embedded therein. Afterward,the smartphone app may either retrieve the cluster-license-info from thepeer-node 152 based on the locator/identifier decoded from the QR code310, or retrieve the cluster-license-info directly from the QR code 310.If the license review device 115 is unable to access the QR code 310from the peer-node 152, unable to extract cluster-license-info 120 fromthe peer-node 152, or unable to evaluate the legitimacy of thecluster-license-info 120, it may determine that the peer-node 152 iscorrupted. In this case, the field operator 110 may cease to perform anyfurther support services to the peer-node 152 or the computer cluster,and record and report the issue.

In some embodiments, after retrieving the cluster-license-info 120 fromthe peer-node 152, the license review device 115 may first perform apreliminary validation of the cluster-license-info 120, and thentransmit the cluster-license-info 120 to a remote license validator 130for further validation. During preliminary validation, the licensereview device 115 may evaluate the integrity and the encryption of thecluster-license-info 120 for any sign of corruption or tampering. Ifevidence of corruption or tampering is found, the license review device115 may display a warning to the field operator 110, so that the fieldoperator 110 may access another peer-node to retrieve thecluster-license-info 120. Further, the field operator 110 may cease toperform any on-site services when a legitimate license in the peer-node152 cannot be found.

In some embodiments, after preliminary validation, the license reviewdevice 115 may transmit (142) the cluster-license-info 120 to thelicense validator 130 for further in-depth validation. The licensevalidator 130 may contain a decryption master key, which can decrypt allthe peer-license-info contained in the cluster-license-info 120.Afterward, the license validator 130 may evaluate thelicense-configurations and the license-usage-data contained in thepeer-license-info, and determine whether there are any licenseviolations in the computer clusters. Alternatively, the smartphone appinstalled on the license review device 115 may contain master keys todecrypt the encrypted cluster-license-info.

In some embodiments, the license validator 130 may identify signs ofsoftware misusages (e.g., without proper licenses or after theexpiration of the licenses), and detect license abuses (e.g., usagesbeyond the device-based, user-based, group-based licenses, hackedlicenses), based on the license configurations and thelicense-usage-data embedded in the cluster-license-info. Alternatively,the license validator 130 may validate the cluster-license-info based onlicense configurations that it maintains. For example, if a cluster nodeis required to register its licenses associated with the softwareinstalled thereon, the license validator 130 may locally store thelicense configuration for the cluster node. During validation, thelicense validator 130 may retrieve the license-configuration for thecluster node from its storage, and compare it with thelicense-usage-data associated with this cluster node.

In some embodiments, the license validator 130 may aggregate thesoftware usage statuses from multiple peer-nodes, and determine whethera multi-user or multi-device type of license is violated. For example,assuming a license can be simultaneously installed and used on up tothree cluster nodes, if the license validator 130 processes thepeer-license-info from all the cluster nodes and identifies that thereare more than three cluster nodes simultaneously installed with thislicense, then the license validator 130 may determine that alicense-misuse violation occurred in the computer cluster.

In some embodiments, the license validator 130 may generate a revisedcluster-license-info 330 based on the license violations detected fromthe cluster-license-info. Specifically, the “revisedcluster-license-info” 330 may include one or more enforcement actionsdesignated to certain cluster nodes with license violations. An“enforcement action” may be a license-related operation to be performedon a specific cluster node, such as activating/deactivating a license,removing a license, changing the type of license, adjusting number oflicenses, or uninstalling the software. For example, when the licensevalidator 130 identifies a multi-device license violation that affectsseveral cluster nodes, it may generate a set of enforcement actionsincluding deactivating or removing the licenses from some cluster nodes,and changing the type of license from multi-device to device-based. Thelicense validator 130 may then merge the enforcement actions into therevised cluster-license-info 330.

In some embodiments, the license validator 130 may return (142) thevalidation outcomes and the revised cluster-license-info to the licensereview device 115. The license review device 115 may then transmit (320)the revised cluster-license-info 330 to the peer-node 152. Afterward,the peer-node 152 may further transmit (331) the revisedcluster-license-info 330 to the core-node 154, which may then distributethe revised cluster-license-info 330 to all the cluster nodes 150 in thecomputer cluster 140.

In some embodiments, after receiving the revised cluster-license-info330, each of the cluster nodes 150 may process the portions of therevised cluster-license-info 330 that is directed to itself.Specifically, for each cluster node 150, its license agent may extractfrom the revised cluster-license-info 330 the enforcement actionscorresponding to the cluster node 150, and then perform theseenforcement actions accordingly. For example, the license agent mayactivate/deactivate licenses, or remove/terminate illegal licenses, oreven uninstall software based on the enforcement actions. After all thecluster nodes 150 received and processed the revisedcluster-license-info 330, the core-node 154 may initiate another roundcluster-initiation process (as described in FIG. 2A) to collect andpropagate cluster-wise license information.

FIG. 4 shows a flow diagram illustrating a process for replicating andenforcing license information in a computer cluster, according to one ormore embodiments of the present disclosure. The processes 401 may setforth various functional blocks or actions that may be described asprocessing steps, functional operations, events, and/or acts, which maybe performed by hardware, software, and/or firmware. Those skilled inthe art in light of the present disclosure will recognize that numerousalternatives to the functional blocks shown in FIG. 5 may be practicedin various implementations.

One skilled in the art will appreciate that, for this and otherprocesses and methods disclosed herein, the functions performed in theprocesses and methods may be implemented in differing order.Furthermore, the outlined steps and operations are only provided asexamples, and some of the steps and operations may be optional, combinedinto fewer steps and operations, or expanded into additional steps andoperations without detracting from the essence of the disclosedembodiments. Moreover, one or more of the outlined steps and operationsmay be performed in parallel.

At block 410, a computer cluster may include a plurality of clusternodes. A core-node may be elected from the plurality of cluster nodes,and those plurality of cluster nodes that are not elected as thecore-node may be deemed a plurality of peer-nodes. The core-node maybroadcast a collection-request message to each of the plurality ofpeer-nodes.

At block 420, the core-node may receive a plurality of peer-license-infofrom the plurality of peer-nodes. Specifically, each of the plurality ofpeer-license-info includes license-info collected by and associated witha corresponding one of the plurality of peer-nodes. In some embodiments,in response to a determination that no peer-license-info is receivedfrom a first peer-node selected from the plurality of peer-nodes, thecore-node may retransmit the collection-request message to the firstpeer-node, and await for the first peer-node to respond with itscorresponding peer-license-info.

At block 430, the core-node may generate a first cluster-license-infobased on the plurality of peer-license-info. In some embodiments, thefirst cluster-license-info may also include a correspondingpeer-license-info collected by the core-node.

At block 440, the core-node may propagate the first cluster-license-infoto each of the plurality of peer-nodes. In some embodiments, the firstcluster-license-info is configured for detecting license violationsamong the plurality of cluster nodes. Specifically, a license validatormay extract from the first cluster-license-info a license-configurationand a license-usage-data associated with specific software. The licensevalidator may then detect license violations by evaluating thelicense-usage-info based on the license configuration.

At block 450, in response to a license-update event from a secondpeer-node selected from the plurality of peer-nodes, the core-node maytransmit the collection-request message to the second peer-node.Specifically, the license-update event may be a new peer-node event, alicense activation event, a license termination event, or a licenseviolation event.

At block 452, the core-node may receive a second peer-license-info fromthe second peer-node. At block 454, the core-node may generate a secondcluster-license-info by adjusting the first cluster-license-info basedon the second peer-license-info. At block 456, the core-node maypropagate the second cluster-license-info to each of the plurality ofpeer-nodes.

At block 460, in response to a license-remove event from a thirdpeer-node selected from the plurality of peer-nodes, the core-node maygenerate a third cluster-license-info by removing peer-license-infoassociated with the third peer-node from the cluster-license-info. Atblock 462, the core-node may propagate the third cluster-license-info toeach of the plurality of peer-nodes.

At block 470, the core-node may receive a revised cluster-license-infogenerated based on the first cluster-license-info. In some embodiments,the license validator may generate the revised cluster-license-info byevaluating the first cluster-license-info for any license violations.The revised cluster-license-info may contain a set of enforcementactions.

At block 472, the core-node may propagate the revisedcluster-license-info to each of the plurality of peer-nodes. Each of theplurality of peer-nodes may be configured to adjust its correspondinglicense-configuration based on the enforcement actions in the revisedcluster-license-info. At block 474, in response to a determination thatthe corresponding license-configuration indicating a license violation,a peer-node may perform the enforcement actions to deactivate itssoftware.

Thus, systems and methods for replicating and enforcing licenseinformation in a computer cluster have been disclosed. The variousembodiments described herein may employ various computer-implementedoperations involving data stored in computer systems. For example, theseoperations may require physical manipulation of physical quantitiesusually, though not necessarily, these quantities may take the form ofelectrical or magnetic signals where they, or representations of them,are capable of being stored, transferred, combined, compared, orotherwise manipulated. Further, such manipulations are often referred toin terms, such as producing, identifying, determining, or comparing. Anyoperations described herein that form part of one or more embodiments ofthe disclosure may be useful machine operations.

In addition, one or more embodiments of the disclosure also relate to adevice or an apparatus for performing these operations. The apparatusmay be specially constructed for specific required purposes, or it maybe a general purpose computer selectively activated or configured by acomputer program stored in the computer. In particular, various generalpurpose machines may be used with computer programs written inaccordance with the teachings herein, or it may be more convenient toconstruct a more specialized apparatus to perform the requiredoperations. The various embodiments described herein may be practicedwith other computer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers, and the like.

One or more embodiments of the present disclosure may be implemented asone or more computer programs or as one or more computer program modulesembodied in one or more computer readable media. The term non-transitorycomputer readable storage medium refers to any data storage device thatcan store data which can thereafter be input to a computer system.Computer readable media may be based on any existing or subsequentlydeveloped technology for embodying computer programs in a manner thatenables them to be read by a computer. Examples of a computer readablemedium include a hard drive, network attached storage (NAS), read-onlymemory, random-access memory (e.g., a flash memory device), a CD(Compact Discs) CD-ROM, a CD-ft or a CD-RW, a DVD (Digital VersatileDisc), a magnetic tape, and other optical and non-optical data storagedevices. The computer readable medium can also be distributed over anetwork coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although one or more embodiments of the present disclosure have beendescribed in some detail for clarity of understanding, it will beapparent that certain changes and modifications may be made within thescope of the claims. Accordingly, the described embodiments are to beconsidered as illustrative and not restrictive, and the scope of theclaims is not to be limited to details given herein, but may be modifiedwithin the scope and equivalents of the claims. In the claims, elementsand/or steps do not imply any particular order of operation, unlessexplicitly stated in the claims.

Plural instances may be provided for components, operations orstructures described herein as a single instance. Finally, boundariesbetween various components, operations and data stores are somewhatarbitrary, and particular operations are illustrated in the context ofspecific illustrative configurations. Other allocations of functionalityare envisioned and may fall within the scope of the disclosure(s). Ingeneral, structures and functionality presented as separate componentsin exemplary configurations may be implemented as a combined structureor component. Similarly, structures and functionality presented as asingle component may be implemented as separate components. These andother variations, modifications, additions, and improvements may fallwithin the scope of the appended claims(s).

In addition, while described virtualization methods have generallyassumed that virtual machines present interfaces consistent with aparticular hardware system, persons of ordinary skill in the art willrecognize that the methods described may be used in conjunction withvirtualizations that do not correspond directly to any particularhardware system. Virtualization systems in accordance with the variousembodiments, implemented as hosted embodiments, non-hosted embodiments,or as embodiments that tend to blur distinctions between the two, areall envisioned. Furthermore, various virtualization operations may bewholly or partially implemented in hardware. For example, a hardwareimplementation may employ a look-up table for modification of storageaccess requests to secure non-disk data.

Many variations, modifications, additions, and improvements arepossible, regardless of the degree of virtualization. The virtualizationsoftware can therefore include components of a host, console, or guestoperating system that performs virtualization functions. Pluralinstances may be provided for components, operations or structuresdescribed herein as a single instance. Finally, boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of the disclosure(s). Ingeneral, structures and functionality presented as separate componentsin exemplary configurations may be implemented as a combined structureor component. Similarly, structures and functionality presented as asingle component may be implemented as separate components. These andother variations, modifications, additions, and improvements may fallwithin the scope of the appended claims(s).

1. A method for evaluating license information in a computer cluster,the method comprising: retrieving, by a license review device, acluster-license-info from a peer-node selected from a plurality ofcluster nodes in the computer cluster, wherein the cluster-license-infocontains a plurality of peer-license-info collected from the pluralityof cluster nodes; receiving, by the license review device, a revisedcluster-license-info generated by detecting license violations in thecluster-license-info; and transmitting, by the license review device,the revised cluster-license-info to the peer-node.
 2. The method ofclaim 1, further comprising: retrieving, by the license review device,the cluster-license-info by scanning a QR code generated by thepeer-node; and decrypting, by the license review device, thecluster-license-info based on a decryption key associated with thecomputer cluster.
 3. The method of claim 1, further comprising:evaluating, by the license review device, the cluster-license-info forany license violations; and in response to a detection of the licenseviolations, generating, by the license review device, a warning for thelicense violations.
 4. The method of claim 1, further comprising:transmitting, by the license review device, the cluster-license-info toa license validator, wherein the license validator generates the revisedcluster-license-info by evaluating the cluster-license-info for any ofthe license violations.
 5. The method of claim 2, wherein the revisedcluster-license-info includes enforcement actions for adjusting licenseconfigurations of the plurality of cluster nodes.
 6. A non-transitorycomputer-readable storage medium, containing a set of instructionswhich, in response to execution by a processor of a license reviewdevice, cause the processor to perform a method for evaluating licenseinformation in a computer cluster, the method comprising: retrieving, bythe license review device, a cluster-license-info from a peer-nodeselected from a plurality of cluster nodes in the computer cluster,wherein the cluster-license-info contains a plurality ofpeer-license-info collected from the plurality of cluster nodes;receiving, by the license review device, a revised cluster-license-infogenerated by detecting license violations in the cluster-license-info;and transmitting, by the license review device, the revisedcluster-license-info to the peer-node.
 7. The non-transitorycomputer-readable storage medium of claim 6, wherein the method furthercomprising: retrieving, by the license review device, thecluster-license-info by scanning a QR code generated by the peer-node;and decrypting, by the license review device, the cluster-license-infobased on a decryption key associated with the computer cluster.
 8. Thenon-transitory computer-readable storage medium of claim 6, wherein themethod further comprising: evaluating, by the license review device, thecluster-license-info for any license violations; and in response to adetection of the license violations, generating, by the license reviewdevice, a warning for the license violations.
 9. The non-transitorycomputer-readable storage medium of claim 6, wherein the method furthercomprising: transmitting, by the license review device, thecluster-license-info to a license validator, wherein the licensevalidator generates the revised cluster-license-info by evaluating thecluster-license-info for any of the license violations.
 10. Thenon-transitory computer-readable storage medium of claim 7, wherein therevised cluster-license-info includes enforcement actions for adjustinglicense configurations of the plurality of cluster nodes.
 11. A licensereview device for evaluating license information in a computer cluster,comprising: a processor; and a non-transitory computer-readable storagemedium, containing a set of instructions which, in response to executionby the processor, cause the processor to perform retrieving acluster-license-info from a peer-node selected from a plurality ofcluster nodes in the computer cluster, wherein the cluster-license-infocontains a plurality of peer-license-info collected from the pluralityof cluster nodes; receiving a revised cluster-license-info generated bydetecting license violations in the cluster-license-info; andtransmitting the revised cluster-license-info to the peer-node.
 12. Thelicense review device of claim 11, wherein the non-transitorycomputer-readable storage medium contains additional instructions which,in response to execution by the processor, cause the processor toperform: retrieving, by the license review device, thecluster-license-info by scanning a QR code generated by the peer-node;and decrypting, by the license review device, the cluster-license-infobased on a decryption key associated with the computer cluster.
 13. Thelicense review device of claim 11, wherein the non-transitorycomputer-readable storage medium contains additional instructions which,in response to execution by the processor, cause the processor toperform: evaluating, by the license review device, thecluster-license-info for any license violations; and in response to adetection of the license violations, generating, by the license reviewdevice, a warning for the license violations.
 14. The license reviewdevice of claim 11, wherein the non-transitory computer-readable storagemedium contains additional instructions which, in response to executionby the processor, cause the processor to perform: transmitting, by thelicense review device, the cluster-license-info to a license validator,wherein the license validator generates the revised cluster-license-infoby evaluating the cluster-license-info for any of the licenseviolations.
 15. The license review device of claim 12, wherein therevised cluster-license-info includes enforcement actions for adjustinglicense configurations of the plurality of cluster nodes.